{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyper-Parameter Tuning with Kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameter optimization or tuning chooses a set of optimal hyperparameters, parameters that control the learning process, for a learning algorithm. The set of hyperparameters yield an optimal model that minimizes a pre-defined loss function on given test data. \n",
    "\n",
    "There are many approaches for HPO: \n",
    "- grid search\n",
    "- random search\n",
    "- bayesian optimization\n",
    "- gradient-based optimization\n",
    "- evolutionary optimization\n",
    "- population based training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Katib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Katib](https://github.com/kubeflow/katib) project is inspired by the [Google Vizier Paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf). \n",
    "\n",
    "Katib is a scalable and flexible hyperparameter tuning framework and is tightly integrated with Kubernetes. It does not depend on any specific deep learning framework (such as TensorFlow, MXNet, or PyTorch).\n",
    "\n",
    "Here are some notes on Katib:\n",
    "* Optimizes a given objective metric such as validation accuracy \n",
    "* Supports Int, Double, Discrete, and Categorical parameter ranges\n",
    "* Option for early stopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyper-Parameter Tuning Examples "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Kubeflow Katib, we will run popular hyper-parameter tuning algorithms including `random search`, `grid search`, and `bayesian optimization`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize ./hyper-parameter-tuning/random-search-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f ./hyper-parameter-tuning/random-search-example.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check manifest, you will see\n",
    "\n",
    "\n",
    "```yaml\n",
    "parameters:\n",
    "- name: --lr\n",
    "  parameterType: double\n",
    "  feasibleSpace:\n",
    "    min: \"0.01\"\n",
    "    max: \"0.03\"\n",
    "- name: --num-layers\n",
    "  parameterType: int\n",
    "  feasibleSpace:\n",
    "    min: \"2\"\n",
    "    max: \"5\"\n",
    "- name: --optimizer\n",
    "  parameterType: categorical\n",
    "  feasibleSpace:\n",
    "    list:\n",
    "    - sgd\n",
    "    - adam\n",
    "    - ftrl\n",
    "```\n",
    "\n",
    "\n",
    "This job generates 3 hyperparameters, parameter type and range are also listed.\n",
    "\n",
    "* --lr (Learning Rate) - type: double\n",
    "* --num-layers (Number of NN Layer) - type: int\n",
    "* --optimizer (optimizer) - type: categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The demo should start an experiment and run three jobs with different parameters. You can run following command to check job status.\n",
    "\n",
    "When the `spec.Status.Condition` changes to Completed, the experiment is finished.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment random-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigate to Katib to Monitor the Hyper-Parameter Tuning Jobs\n",
    "\n",
    "You can monitor your results in the Katib UI. If you installed Kubeflow using the deployment guide, you can access the Katib UI at `https://<your kubeflow endpoint>/katib/`\n",
    "\n",
    "For `random-experiment`, please go to `HP (HypterParameter)` -> `Monitor` -> `random-experiment`.\n",
    "\n",
    "![katib-experiment-selection.png](./img/katib-experiment-selection.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick up best parameters in from results\n",
    "\n",
    "Once you click job and go the detail page, you will see different combination of parameters and accuracy.\n",
    "\n",
    "\n",
    "| trialName  | Validation-accuracy \t| accuracy \t| --lr \t| --num-layers \t| --optimizer|\n",
    "|----------------------------|----------|----------|----------------------|---|------|\n",
    "| random-experiment-rfwwbnsd | 0.974920 | 0.984844 | 0.013831565266960293 | 4 | sgd  |\n",
    "| random-experiment-vxgwlgqq | 0.113854 | 0.116646 | 0.024225789898529138 | 4 | ftrl |\n",
    "| random-experiment-wclrwlcq | 0.979697 | 0.998437 | 0.021916171239020756 | 4 | sgd  |\n",
    "| random-experiment-7lsc4pwb | 0.113854 | 0.115312 | 0.024163810384272653 | 5 | ftrl |\n",
    "| random-experiment-86vv9vgv | 0.963475 | 0.971562 | 0.02943228249244735  | 3 | adam |\n",
    "| random-experiment-jh884cxz | 0.981091 | 0.999219 | 0.022372025623908262 | 2 | sgd  |\n",
    "| random-experiment-sgtwhrgz | 0.980693 | 0.997969 | 0.016641686851083654 | 4 | sgd  |\n",
    "| random-experiment-c6vvz6dv | 0.980792 | 0.998906 | 0.0264125850165842   | 3 | sgd  |\n",
    "| random-experiment-vqs2xmfj | 0.113854 | 0.105313 | 0.026629394628228185 | 4 | ftrl |\n",
    "| random-experiment-bv8lsh2m | 0.980195 | 0.999375 | 0.021769570793012488 | 2 | sgd  |\n",
    "| random-experiment-7vbnqc7z | 0.113854 | 0.102188 | 0.025079750575740783 | 4 | ftrl |\n",
    "| random-experiment-kwj9drmg | 0.979498 | 0.995469 | 0.014985919312945063 | 4 | sgd  |\n",
    "\n",
    "\n",
    "![katib-experiment-result.png](./img/katib-experiment-result.png)\n",
    "\n",
    "You can also click trail name to check Trial data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: All rest examples are different optimization algorithms.  \n",
    "> The way to submit the job and check job lifecycle is same as random-search-example we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize ./hyper-parameter-tuning/grid-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f ./hyper-parameter-tuning/grid-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment grid-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayesian\n",
    "\n",
    "BayesOpt: A toolbox for bayesian optimization, experimental design and stochastic bandits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize ./hyper-parameter-tuning/bayesopt-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create -f ./hyper-parameter-tuning/bayesopt-example.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe experiment bayesopt-example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigate to Katib to Monitor All Hyper-Parameter Tuning Jobs\n",
    "![katib-experiment-selection.png](./img/katib-experiment-selection.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
